* Added `lazy_loader` to known list ([#2991](#2991)). With this commit, the `lazy_loader` module has been added to the known list in the configuration file, addressing a portion of issue [#193](#193), which may have been caused by the discovery or loading of this module. The `lazy_loader` is a package or module that, once added to the known list, will be recognized and loaded by the system. This change does not affect any existing functionality or introduce new methods. The commit solely updates the known.json file to include `lazy_loader` with an empty list, indicating that it is ready for use. This modification will enable the correct loading and recognition of the `lazy_loader` module in the system.
* Added `librosa` to known list ([#2992](#2992)). In this update, we have added several open-source libraries to the known list in the configuration file, including `librosa`, `llvmlite`, `msgpack`, `pooch`, `soundfile`, and `soxr`. These libraries are commonly used in data engineering, machine learning, and scientific computing tasks. `librosa` is a Python library for audio and music analysis, while `llvmlite` is a lightweight Python interface to the LLVM compiler infrastructure. `msgpack` is a binary serialization format like JSON, `pooch` is a package for managing external data files, `soundfile` is a library for reading and writing audio files, and `soxr` is a library for high-quality audio resampling. Each library has an empty list next to it for specifying additional configuration related to the library. This update partially resolves issue [#1931](#1931) by adding `librosa` to the known list, ensuring that these libraries will be properly recognized and utilized by the codebase.
* Added `linkify-it-py` to known list ([#2993](#2993)). In this release, we have added support for two new open-source packages, `linkify-it-py` and `uc-micro-py`, to enhance the software's functionality and compatibility. The addition of `linkify-it-py` and its constituent modules, as well as the incorporation of `uc-micro-py` with its modules and classes, aims to expand the software's capabilities. These changes are related to the resolution of issue [#1931](#1931), and they will enable the software to work seamlessly with these packages, thereby providing a better user experience.
* Added `lz4` to known list ([#2994](#2994)). In this release, we have added support for the LZ4 lossless data compression algorithm, which is known for its focus on compression and decompression speed. The implementation includes four variants: lz4, lz4.block, lz4.frame, and lz4.version, each providing different levels of compression and decompression speed and flexibility. This addition expands the range of supported compression algorithms, providing more options for users to choose from and partially addressing issue [#1931](#1931) related to supporting additional compression algorithms. This improvement will be beneficial to software engineers working with data compression in their projects.
* Fixed `SystemError: AST constructor recursion depth mismatch` failing the entire job ([#3000](#3000)). This PR introduces more deterministic, Go-style, error handling for parsing Python code, addressing issues that caused the entire job to fail due to a `SystemError: AST constructor recursion depth mismatch` ([#3000](#3000)) and bug [#2976](#2976). It includes removing the `AstroidSyntaxError` import, adding an import for `SqlglotError`, and updating the `SqlParseError` exception to `SqlglotError` in the `lint` method of the `SqlLinter` class. Additionally, abstract classes `TablePyCollector` and `DfsaPyCollector` and their respective methods for collecting tables and direct file system accesses have been removed. The `PythonSequentialLinter` class, previously handling multiple responsibilities, has also been removed, enhancing code modularity, understandability, maintainability, and testability. The changes affect the `base.py`, `python_ast.py`, and `python_sequential_linter.py` modules.
* Skip applying permissions for workspace system groups to Unity Catalog resources ([#2997](#2997)). This commit introduces changes to the ACL-related code in the `databricks labs ucx create-catalog-schemas` command and the `migrate-table-*` workflow, skipping the application of permissions for workspace system groups in the Unity Catalog. These system groups, which include 'admins', do not exist at the account level. To ensure the correctness of these modifications, unit and integration tests have been added, including a test that checks the proper handling of user privileges in system groups during catalog schema creation. The `AccessControlResponse` object has been updated for the `admins` and `users` groups, granting them specific permissions for a workspace and warehouse object, respectively, enhancing the system's functionality in multi-user environments with system groups.

* Added `lazy_loader` to known list
([#2991](#2991)). With this
commit, the `lazy_loader` module has been added to the known list in the
configuration file, addressing a portion of issue
[#193](#193), which may have
been caused by the discovery or loading of this module. The
`lazy_loader` is a package or module that, once added to the known list,
will be recognized and loaded by the system. This change does not affect
any existing functionality or introduce new methods. The commit solely
updates the known.json file to include `lazy_loader` with an empty list,
indicating that it is ready for use. This modification will enable the
correct loading and recognition of the `lazy_loader` module in the
system.
* Added `librosa` to known list
([#2992](#2992)). In this
update, we have added several open-source libraries to the known list in
the configuration file, including `librosa`, `llvmlite`, `msgpack`,
`pooch`, `soundfile`, and `soxr`. These libraries are commonly used in
data engineering, machine learning, and scientific computing tasks.
`librosa` is a Python library for audio and music analysis, while
`llvmlite` is a lightweight Python interface to the LLVM compiler
infrastructure. `msgpack` is a binary serialization format like JSON,
`pooch` is a package for managing external data files, `soundfile` is a
library for reading and writing audio files, and `soxr` is a library for
high-quality audio resampling. Each library has an empty list next to it
for specifying additional configuration related to the library. This
update partially resolves issue
[#1931](#1931) by adding
`librosa` to the known list, ensuring that these libraries will be
properly recognized and utilized by the codebase.
* Added `linkify-it-py` to known list
([#2993](#2993)). In this
release, we have added support for two new open-source packages,
`linkify-it-py` and `uc-micro-py`, to enhance the software's
functionality and compatibility. The addition of `linkify-it-py` and its
constituent modules, as well as the incorporation of `uc-micro-py` with
its modules and classes, aims to expand the software's capabilities.
These changes are related to the resolution of issue
[#1931](#1931), and they
will enable the software to work seamlessly with these packages, thereby
providing a better user experience.
* Added `lz4` to known list
([#2994](#2994)). In this
release, we have added support for the LZ4 lossless data compression
algorithm, which is known for its focus on compression and decompression
speed. The implementation includes four variants: lz4, lz4.block,
lz4.frame, and lz4.version, each providing different levels of
compression and decompression speed and flexibility. This addition
expands the range of supported compression algorithms, providing more
options for users to choose from and partially addressing issue
[#1931](#1931) related to
supporting additional compression algorithms. This improvement will be
beneficial to software engineers working with data compression in their
projects.
* Fixed `SystemError: AST constructor recursion depth mismatch` failing
the entire job
([#3000](#3000)). This PR
introduces more deterministic, Go-style, error handling for parsing
Python code, addressing issues that caused the entire job to fail due to
a `SystemError: AST constructor recursion depth mismatch`
([#3000](#3000)) and bug
[#2976](#2976). It includes
removing the `AstroidSyntaxError` import, adding an import for
`SqlglotError`, and updating the `SqlParseError` exception to
`SqlglotError` in the `lint` method of the `SqlLinter` class.
Additionally, abstract classes `TablePyCollector` and `DfsaPyCollector`
and their respective methods for collecting tables and direct file
system accesses have been removed. The `PythonSequentialLinter` class,
previously handling multiple responsibilities, has also been removed,
enhancing code modularity, understandability, maintainability, and
testability. The changes affect the `base.py`, `python_ast.py`, and
`python_sequential_linter.py` modules.
* Skip applying permissions for workspace system groups to Unity Catalog
resources ([#2997](#2997)).
This commit introduces changes to the ACL-related code in the
`databricks labs ucx create-catalog-schemas` command and the
`migrate-table-*` workflow, skipping the application of permissions for
workspace system groups in the Unity Catalog. These system groups, which
include 'admins', do not exist at the account level. To ensure the
correctness of these modifications, unit and integration tests have been
added, including a test that checks the proper handling of user
privileges in system groups during catalog schema creation. The
`AccessControlResponse` object has been updated for the `admins` and
`users` groups, granting them specific permissions for a workspace and
warehouse object, respectively, enhancing the system's functionality in
multi-user environments with system groups.